A Neural Clustering Algorithm for Estimating Visible Articulatory Trajectory

The bimodal acoustic-visual nature of speech establishes sound correlations between its audio component and the corresponding articulatory information associated to the time-varying geometry of the vocal tract. In this paper we propose an estimation structure consisting of a simpliied Time-Delay Neural Network (TDNN) working on 4-5 dimensional cepstrum trajectories provided by a preceding clusterization layer based on a Self Organizing Map (SOM). The use of this pre-processing layer has allowed an eeective non-linear clusterization of cepstrum vectors thus simplifying of one order the complexity of the resulting system while maintaining unchanged the global estimation performances. The achieved results are shown in terms estimation precision and robustness with reference to previously published results. 1 A direct approach to articulatory estimation The objective of any direct approach to articulatory estimation is the design of a suitable mechanism for mapping a predeened acoustic representation of speech into a corresponding motor space representation. No intermediate explicit recognition or classiication is required since it is assumed that all the necessary processing is embedded in the conversion mechanism itself. Extensive experimentations on normal hearing and hearing impaired subjects 2] 3] have clearly demonstrated that if, on one hand, phonemes can be associated rather easily to well deened mouth conngurations (called "visemes"), the inverse association is usually troublesome since the same posture of the mouth can correspond to diierent phonemes. As an example the "bilabial" viseme is associated to diierent phonemes like /m,p,b/ and the "velar" viseme is associated to diierent phonemes like /k,g/. Moreover, deep investigation on the articulatory dynamics 4] 5] stress the role played by the coarticulatory phenomena which describe the eeects on articulation due to past acoustic outputs (backward coarticulation) and to future going-to-be-produced acoustic information (forward coarticulation). Since no exhaustive study on coarticulatory dynamics is still available, an approach to speech articulatory description passing through the phonemic level results extremely complex and, at least, formally incomplete. On the contrary, a more viable solution seems that of performing the articulatory estimation